Large in-memory databases are currently being used on production bases to serve as key-value stores, online transaction processing (OLTP) databases, large-scale web applications, massive online multi-player games, software defined networks (SDN) controller network views, scientific simulation, etc. These applications are built to sustain high transaction rates using multiple execution contexts, e.g., threads in non-uniform memory access (NUMA) shared-everything architectures, whereby the dataset is logically partitioned and each partition is associated with an execution context which is responsible to process transactions on its partition.
Checkpointing and logical redo log may be used in in-memory datasets to enable dataset recovery in case of system failure. The durability of update transactions is provided by a periodic checkpointing process that flushes a consistent instantaneous state to a persistent storage, and a logical redo log that records all the modification requests between consecutive checkpoints. After a system failure, a restore process uses the latest persistent checkpoint to initialize the memory state, and a replay procedure to replay the redo log to re-create the application state at the point before the time of the failure.
The design of the redo-logging, in face of the dataset high update rates, remains a continuing challenge for the database designers. Redo-logging is tightly coupled with the checkpointing. The checkpointing process must be low-overhead and highly efficient in order to support high update rates, and the frequent checkpointing in order to reduce the redo log size and its associated log replay duration at restore-time. Since checkpoints run repeatedly, an incremental checkpointing may usually be utilized whereby only objects changed since the previous checkpoint are detected and persisted.
Logical redo logging usually persists the modification requests to the dataset while these modifications take effect in the in-memory data set. As the rate of updating a persistent storage device is usually significantly lower than updating memory, there may be a tension between slowing down the transaction processing in favor of persisting a more complete log of these transactions.
Tuan Cao et al.: “Fast Checkpoint Recovery Algorithms for Frequently Consistent Applications”, SIGMOD′11. June 2011, Athens, Greece describes an approach utilizing special-purpose storage devices in favor of enabling the logging of transaction in full memory speed.